Informativeness for Adhoc IR Evaluation: A Measure that Prevents Assessing Individual Documents
نویسندگان
چکیده
Informativeness measures have been used in interactive information retrieval and automatic summarization evaluation. Indeed, as opposed to adhoc retrieval, these two tasks cannot rely on the Cranfield evaluation paradigm in which retrieved documents are compared to static query relevance document lists. In this paper, we explore the use of informativeness measures to evaluate adhoc task. The advantage of the proposed evaluation framework is that it does not rely on an exhaustive reference and can be used in a changing environment in which new documents occur, and for which relevance has not been assessed. We show that the correlation between the official system ranking and the informativeness measure is specifically high for most of the TREC adhoc tracks.
منابع مشابه
Using Factual Density to Measure Informativeness of Web Documents
The information obtained from the Web is increasingly important for decision making and for our everyday tasks. Due to the growth of uncertified sources, blogosphere, comments in the social media and automatically generated texts, the need to measure the quality of text information found on the Internet is becoming of crucial importance. It has been suggested that factual density can be used to...
متن کاملSynonym-Based Expansion and Boosting-Based Re-Ranking: A Two-phase Approach for Genomic Information Retrieval
We describe in this paper the design and evaluation of the system built at Simon Fraser University for the TREC 2005 adhoc retrieval task in the Genomics track. The main approach taken in our system was to expand synonyms by exploiting a fusion of a set of biomedical and general ontology sources, and apply machine learning and natural language processing techniques to re-rank retrieved document...
متن کاملTweet Contextualization (Answering Tweet Question) - the Role of Multi-document Summarization
The article presents the experiments carried out as part of the participation in the Tweet Contextualization (TC) track of INEX 2013. In our system there are three major sub-systems; i) Offline multi-document summarization, ii) Focused IR and iii) online multi-document Summarization. The Offline multi-document summarization system is based on document graph, clustering and sentence compression....
متن کاملFinancial Statement Comparability: Benets and Costs
This paper analyzes the bene ts and costs of nancial statement comparability from an information perspective. We rst show analytically that while comparability enhances the correlation among rmsreported earnings (common informativeness), it reduces rmsown reporting precision (individual informativeness). Next, we provide theoretical justi cation for the comparability measure of De Fra...
متن کاملEvaluating Clusterings by Estimating Clarity
In this thesis I examine clustering evaluation, with a subfocus on text clusterings specifically. The principal work of this thesis is the development, analysis, and testing of a new internal clustering quality measure called informativeness. I begin by reviewing clustering in general. I then review current clustering quality measures, accompanying this with an in-depth discussion of many of th...
متن کامل